As we've talked about in the past, the focus on data – how much is being generated, where it's being created, the tools needed to take advantage of it, the shortage of skilled talent to manage it, and so on – is rapidly changing the way enterprises are operating both in the datacenter and in the cloud and dictating many of the product roadmaps being developed by tech vendors. Automation, analytics, artificial intelligence (AI) and machine learning, and the ability to easily move applications and data between on-premises and cloud environments are the focus of much of what OEMs and other tech players are doing. And all of this is being accelerated by the COVID-19 pandemic, which is speeding up enterprise movement to the cloud and forcing them to adapt to a suddenly widely distributed workforce, trends that won't be changing any time soon as the coronavirus outbreak tightens its grip, particularly in the United States. OEMs over the past several months have been particularly aggressive in expanding their offerings in the storage sector, which is playing a central role in help enterprises bridge the space between the datacenter, the cloud and the network edge and to deal with the vast amounts of structured and – in particular – unstructured data being created. That can be seen in announcements that some of the larger vendors have made over the past few months.

There is no doubt that the AI marketplace is an attractive, fast-growing sector for IBM Business Partners to get engaged. IDC has forecasted the 2020 AI revenue opportunity at USD 49 billion, growing to USD 98 billion by 2023¹. Customers tell us that infrastructure for AI is not getting better, it’s getting worse. In their struggle to help meet escalating demands, many vendors resort to cobbling disparate portfolios that are not tightly integrated. The result is a jigsaw puzzle of data silos, multiple data copies and independent components, which lead to growing complexity with increased cost of ownership and poor data insights. Data locked in silos is complex to manage—it can also increase operating expenses, which, in turn, drives up capital expenses. It is also hard for users to collect, organize and analyze to extract value. AI needs data to achieve the holistic view that is required for meaningful insights that fuels business growth. How can Business…

IBM launched a storage system designed with Nvidia for artificial intelligence workloads and use of data tools such as TensorFlow, PyTorch and Spark. The effort, called IBM Spectrum AI with Nvidia DGX, is a converged system the combines a software-defined file system, all-flash and Nvidia's DGX-1 GPU system. IBM Spectrum AI is part of a broader trend by storage vendors to go all-flash and create architectures better suited for AI workloads and machine learning. Pure Storage and Nvidia had paired up on AI-ready infrastructure with the former targeting a data hub architecture. NetApp, which now aims to be more of a data management player, and Lenovo partnered on AI systems.

In helping dozens of organizations build on-premises AI initiatives, we have seen three fundamental stages organizations go through on their journey to enterprise-scale AI. First, individual data scientists experiment on proof of concept projects which may be promising. These PoCs then often hit knowledge, data management and infrastructure performance obstacles that keep them from proceeding to the second stage to deliver optimized and trained models quickly enough to deliver value to the organization. Moving to the third and final stage of AI adoption, where AI is integrated across multiple lines of business and requires enterprise-scale infrastructure, presents significant integration, security and support challenges. Today IBM introduced IBM PowerAI Enterprise and an on-premises AI infrastructure reference architecture to help organizations jump-start AI and deep learning projects, and to remove the obstacles to moving from experimentation to production and ultimately to enterprise-scale AI.

DDN is upping array capacities and access speeds with one eye on its traditional HPC customer base and the other on businesses that are testing or deploying deep learning architectures. AI, or perhaps more accurately Machine Learning, could be a gift that keeps on giving for storage and systems players as customers try to crunch through ever growing volumes of business data in a decent time. DDN has two parallel file system scale-out arrays – EXAScaler using Lustre and the GRIDScaler using Spectrum Scale (GPFS for historians). It has just bought Intel's Lustre assets, including its development and support teams, and aims to help accelerate Lustre's development, which is lagging compared to IBM's Spectrum Scale. Parallel file systems provide faster access to data than sequential access filer software, particularly as capacities and the number of users rise.

Data is the core of artificial intelligence; without data there is no learning and the more data available to the training systems the better the accuracy. Customers have been utilizing high-performance data storage solutions that were originally built for HPC environments to address the data challenges of machine learning. One of the more prominent high-performance storage solutions is IBM's Spectrum Scale file system. In this blog post I outline five specific shortfalls of Spectrum Scale that limit its ability to meet the demands of AI systems. IBM Spectrum Scale (aka IBM GPFS) was developed 25 years ago to support the high throughput needs of multimedia applications.